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Mandatory Requirement 


Section/Page in Supplier’s Response Met Comments 


The supplier must provide examples of how it | DC/MIT in Joint Venture has provided 
addresses ethical* practices when delivering | examples of how it addresses ethical* 
Al. This includes demonstrating experience in | practices when delivering Al. Including 
applying frameworks, methods, guidelines or | experience in applying frameworks, 
assessment tools to test datasets and |methods, guidelines or assessment tools to 
outcomes. test datasets and outcomes. 


“For purposes of AI, ethical practices which are | Please refer to Section 5, Page 16 
transparent, respect procedural fairness, and 

demonstrate high-quality data practices. This 

includes, but is not limited to, testing for 

outcomes and biases and fair, comprehensive 

and inclusive data collection practices. 
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Public Services and Procurement Canada (PSPC) 
Artificial Intelligence (Al) — Invitation to Qualify (ITQ) 
Solicitation Number: EN578-180001/A 


5. ETHICAL PRACTICES (M3) 


Artificial Intelligence (Al) provides the opportunity to gain a deep understanding of customer data and 
insights. With it comes a number of ethical dilemmas in terms of AI replacing humans, and or in the 
consolidation of tremendous amounts of personal information requiring safeguards to protect privacy and 
confidentiality. Used appropriately, Artificial Intelligence becomes a complimentary service to human 
intelligence and ingenuity to provide a clear understanding of the business environment, market factors, 
customer engagement and customer service requirements. 


DC/MIT in Joint Venture completes our engagements within the AI space under the direction and guidance 
of customers seeking industry specific tool sets and applications to deliver AI based solutions, whether it 
be predictive analytics, natural language processing, etc... In developing these solutions, the requirements 
are closely developed with the customers to ensure that they are developed in accordance with the 
customers privacy and security regulations as a core requirement. Solutions can be developed to meet all 
legislated requirements including PIPEDA, HIPAA and GoC security requirements. 


On each engagement, prior to developing the solutions, DC/MIT in Joint Venture would meet with the client 
representatives to develop the following: 


1. Create a governance framework for the project to develop the requirements for Al and the 
safeguards to be implemented including integrating Al into the technology strategies for the project, 
clearly identifying and mapping all data sets being accessed and consolidated, monitoring and 
testing the Al solution for accuracy and tuning the applications as needed; 

2. Adhere to GoC standards and policies for security, transparency and accountability such as the 
open government initiatives. Developing solutions that users can trust and provide credible results 
is paramount to gaining acceptance and use of applications. DC/MIT in Joint Venture will meet 
high quality data standards, monitor solutions for accuracy and refer to GoC ethics experts for 
advice in developing and operating the solutions; 

3. Develop solutions which are user centric and easy to use, ensuring that customers understand 
how the Al solution uses information and protects their interests. This includes development of 
the user experience to ensure use of simple language, and the development of open and 
transparent systems; and 

4. Conform with GoC security standards in completing Threat and Risk Assessments (TRAs) for each 
solution to clearly define all known threats and provide all required mitigation solutions. 


This framework has been built from the Sage Group’s framework for ethical Al Sources and will be tailored 
as required per engagement and for the Government of Canada’s needs. 


Government Gouvernement 
=| of Canada du Canada 
DC/MIT in Joint Venture - CONFIDENTIAL Page 16 


Divulgué en vertu de la LAI 
Released under the ATIA 


000019 


Pages 20 to / à 24 
are not relevant 


sont non pertinentes 


Divulgué en vertu de la LAI 
Released under the ATIA 


JocuSign Envelope iD: 6A9C6B0C-C06D-4182-9F2B-CE856A8B7F1A |. « ; «SH st 7 LS AA 
| | | LEIC ON Nd8p'ci 8100 “LE ‘110 awl] poalosay 
s.20(1)(c) 


Part A 
Al Ethics — 


At Element AI, ethical AI considerations are of paramount importance to our daily . 

practice. These considerations permeate through all aspects of our internal dedisionsvaking 
and partnership interactions. Our perspective and position on ethics has not only informed: our 
approach to governance, but also ae us to develop pee and tools that aim to. address 
these complex issues. | 


Governance 


We have reflected on our internal position with respect to ethical.Al,.it is one that will be iterated 
on as the AI landscape evolves. At present state, we have a stance on what we will and will not 
use Al for. In areas where we choose to apply Al, we have developed an understanding of the 
risks and direction on how to manage them. For example, we recognize the safety risks and 
mitigation strategies associated with robotics differ from those associated with agricultural Al - 
applications. We use this knowledge to act as advocates amongst our industry peers and public. 
bodies, both locally and internationally for good AI governance. For example, we have recently 
invited as expert commentators on a proposed model Al governance framework for a national 
agency responsible for citizens’ privacy. We have also establ shed, areas where. we will not 
apply AI, such as lethal autonomous weapons. L ae 


Our focus on 1 ethical Al goes well beyond. doing the thinking and simply sharing internally. We 

are actively working to ensure our principles ground our actions: across the broader .. 
organization. From leadership through to team, we are educ ting to make all aware of the risks 
and mitigations, SO that we have the foresight to ide strateg es appropriately. 


This practice of internal education has expanded more | broadly to the community. We are. shout 
to publish a series of articles.on our external blog to. educate the AI community about the. |. 
challenges of AI trustability. The kay: message within this series is that mass adoption of Al in . 
enterprise will require a focus on tackling several issues rega ding trustability of Al. We propose 

a taxonomy to enumerate and characterize these issues which include robustness, fairness, - 
privacy, reproducibility, accountability, and explainability; to note, explainability is identified as. 
the most significant challenge within Al trustability. This series will be accompanied with a white- 
paper dedicated to an in-depth overview of the explainable AI literature. We are working with. 
our clients to ensure they understand the various risks of AI, and can build the appropriate ` 
governance mechanisms to effectively scale oversight acrossia portfolio of Al applications. 


in addition to dai we are building a team that has a specific mandai of AI for Good. As. 
part of our efforts to democratize Al and.make Al accessible for everyone, it’s important: to | 
make sure that we explore ways to use Al to help sociéty. Problems like poverty, climate. 
change, disease, and much more can leverage AI and machine learning to make these issues 
easier to tackle. In that spirit, we opened a lab dedicated to Al for Good earlier this year to work 
with NGOS to bring Al research into the nonprofit world. 


The mandate of Al for Good i is three-fold: | 

1. Empower those who are fighting the good fight. We are partnering with non-g overn ment | 
‘organizations, lending our expertise so that we can co-create initiatives that support the 
growth of social good. Examples include 
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2. Provide methodological research as needed i in the deploy ani of Al-for-Good applications. 
Our efforts in this area help to ensure that our products and systems are safe and inclusive, 
and also include accountability measures so that there are mechanisms to enact when — 
things go wrong, and that human work continues to be me aningful and sufficient. Examples 
of this research include areas such as explainability, fairnéss, bias, causality. aS 

3. Build capacity. at.the point of need. We recognize the opportunity to enable collaboration by 

| leveraging relationships created amongst EAI; NGOs and the corporate world. We believe 

‘in supporting social growth by investing in people, supporting and -building communities | 
globally, as well as creating and disseminating knowledge internationally. Investments we 
make in this respect are thoughtfully placed to maximize i pact. For example, we | 
sponsoréd the participation of high schools students from Ghana to MISE (Mathematics 
inspired Science and Engineering) program last year, as part of our efforts build talent. 
outside of MacuUohabi tech communities. | 


Practices and Tools | 


As we un deploying AI models, it became imperative to de osi an ünderstanding as to how 
and why these models make the decisions that they do. Some people consider AI models today 

to be "black boxes” and are unsatisfied with this state — expla nations are demanded by our ` 
customers and they are sought ad.hoc by our own researchers . Explainabllity will not pe | 
“solved” in the coming years, rather it is a territory to be naviga ited. | 


That is De we now have a dedicated Explainability team. This group provides tools to identify 
and alleviate unwanted bias in Al models during development and in production, to provide | 
guidance on appropriate interpretation of models or explanation of outputs, and to ensure 
models are focusing on reasonable (vs. spurious) correlations! This work is grounded by . 
human-in-the-loop principles. From an ethics standpoint, this means that regulators will have the 
opportunity to evaluate Al models against their ethical require ents. To provide an example of 
potential-future explainability tool applications, consider the circumstance whereby a model 
used to predict lending risk reveals that its predictions disproportionately penalize a certain | © 
race. This issue could be diagnosed through examining explanations that identify the features : 
that influence this prediction the most, and various criteria to enforce fairness could then be- 
APRIRE te to the model to ensure a fair treatment of protected fee tures (i.e. , gender, race, etc.). 


In addition to our focus on explainability, we are also fogüšed on data preparation to ensure 
data quality is strong, and data security to ensure data is guarded from cybersecurity | Lea 
threats: On this note, we are SOC2 compliant. 
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Ethical considerations, 
detined data practices and 
bias testing are 
foundational to our AI 
solutions, including IBM’s 
Principles on Trust and 
Transparency and the 
launch of IBM OpenScale 
AI. Ethical practices, 
methods, guidelines and 
frameworks are 
substantiated through 
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Mandatory Criteria 


M3 - Ethics in AI 


IBM’s Principles for Trust and Transparency in AI 
Artificial intelligence technology is rapidly growing in capability, impact and influence. IBM is dedicated to delivering AI 


services that are built ethically, responsibly, are unbiased and explainable. We are continually working to evaluate and 
update our services and offerings, advancing them in a way that is trustworthy and inclusive. 


As enterprises build and deploy artificial intelligence systems, it is important to understand the ethical considerations of 
our work. Ethics are not a separate business objective bolted on after an AI system has been deployed. They are part of 
business performance. Only by embedding ethical principles into AI applications, frameworks, methods and processes 
can we build systems that people can trust. IBM has for decades followed core principles —grounded in commitments to 
Trust and Transparency—that guide our handling of client data and insights, and also our responsible development, 
deployment and transparent use of artificial intelligence and other transformative innovations. Not only does IBM 
embrace these principles as our “guiding light” in developing the Watson platform and customer AI applications on top 
of it, our CEO, Ginni Rometty, shares this vision and encourages business leaders to follow IBM’s Principles for Trust and 
Transparency: 


1. The purpose of AI is to augment human intelligence 

The purpose of AI and cognitive systems developed and applied by IBM is to augment—not replace—human intelligence. 
Our technology is and will be designed to enhance and extend human capability and potential. At IBM, we believe AI 
should make ALL of us better at our jobs, and that the benefits of the AI era should touch the many, not just the elite few. 
To that end, we are investing in initiatives to help the global workforce gain the skills needed to work in partnership with 
these technologies. IBM believes in the investment in skills for the era of man + machine. 


2. Data and insights belong to their creator 

IBM clients’ data is their data, and their insights are their insights. Client data and the insights produced on IBM’s cloud 
or from IBM’s AI are owned by IBM's clients. We believe that government data policies should be fair and equitable and 
prioritize openness. IBM’s high-quality data practice considers the following views on data and AI: 


Data Ownership: Clients are not required to relinquish rights to their data—or insights derived from it—to have the 


benefits of IBM’s solutions and services. 
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Data Privacy: IBM is fully committed to protecting the privacy of our clients’ data, which is fundamental in a data-driven 
society. 


Data Security: IBM is devoting our powerful engines of innovation to create tools to protect our clients, their data and 
global trade from cyber threats, and convening a broader discussion on balancing security, privacy and freedom. 


Government Access to Data: IBM has not provided client data to any government agency under any surveillance program 
involving bulk collection of content or metadata. 


Cross-Border Data Flows: IBM views the free movement of data across borders as essential to 21st century commerce. 


3; AI Systems must be transparent and explainable 

For the public to trust AI, it must be transparent. Technology companies must be clear about who trains their AI 
systems, what data was used in that training and, most importantly, what went into their algorithm’s recommendations. 
If we are to use AI to help make important decisions, it must be explainable. 


IBM will ALWAYS make clear: 


e When and for what purposes AI is being applied in the cognitive solutions we develop and deploy. 

e The major sources of data and expertise that inform the insights of cognitive solutions, as well as the methods 
used to train those systems and solutions. 

e That while bias can never be fully eliminated, and our work to eliminate it will never be complete, we and all 
companies advancing AI have an obligation to address it proactively. We therefore continually test our systems 
and find new data sets to better align their output with human values and expectations. 

e The principle that clients own their own business models and intellectual property and that they can use AI and 
cognitive systems to enhance the advantages they have built. We will work with our clients to protect their data 
and insights, and will encourage our clients, partners and industry colleagues to adopt similar practices. 

e Ourfirm support for transparency and data governance policies that will ensure people understand how an AI 


system came to a conclusion or recommendation. 


Guided by these principles, here's what IBM is doing to address ethics and bias in AI: 


1. IBMwasoneofthe first companies in the world to adopt a set of principles for trust and transparency for new 
technologies, including AI systems 


IBM has introduced a comprehensive new set of trust and transparency capabilities for AI on the IBM Cloud. This 
software service brings to market new technologies developed by IBM Research together with Watson’s engineering 
team. The capabilities address multiple dimensions of trust to enable AI solutions that inspire contidence: 
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Robustness: IBM is working to ensure the security and reliability of AI systems by exposing and fixing their 
vulnerabilities: identifying new attacks and defense, designing new adversarial training methods to strengthen against 
attack, and developing new metric to evaluate robustness. 


Explainability: Knowing how an AI system arrives at an outcome is key to trust, particularly for enterprise AI. To improve 
transparency, IBM is researching local and global interpretability of models and their output, training for interpretable 
models and visualization of information flow within models, and teaching explanations. 


Fairness: To encourage the adoption of AI, we must ensure it does not take on and amplify our biases. IBM is creating 
methodologies to detect and mitigate bias through the life cycle of AI applications. 


Lineage: Lineage services can infuse trust in AI systems by ensuring all their components and events are trackable. IBM 
is developing services like instrumentation and event generation, scalable event ingestion and management, and 


efficient lineage query services to manage the complete lifecycle of AI systems. 


2. IBM is solving AT's “black box” problem: IBM AI OpenScale 


At IBM, we understand the roadblocks to full AI adoption — from trust and transparency concerns with the ‘black box’ of 
AI, to problems of scale and automation — and we are working with enterprises to provide the tools needed to overcome 
these roadblocks and deliver real business value. To that end, IBM has made a critical move in helping businesses 
accelerate the adoption of AI with the release of IBM AI OpenScale. 


AI OpenScale allows businesses to operate and automate AI at scale and provides businesses with confidence in AI 
decisions. AI OpenScale infuses AI throughout its full lifecycle with trust and transparency and explains outcomes and 
automatically eliminates bias. In addition, the platform enables organizations to keep their options open in terms of AI 
deployment, with a wide range of connectors that allow users to build and deploy their models anywhere - in the IBM 
Cloud, in IBM Cloud Private, or on a variety of other cloud platforms. 
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AI OpenScale provides enterprises visibility into how AI is built, used, and performs. With AI OpenScale, businesses can 
embed AI into new or existing business applications and functions with the freedom to use the environment of their 
choice. 


For Government, it is critical to ensure AI recommendations or decisions are fully traceable. AI OpenScale enables 
enterprises to audit the lineage of the models and the associated training data, along with the inputs and outputs for 
each AI recommendation. Building on these capabilities, AI OpenScale provides explainability for black box models and 
functions, automatic bias detection and mitigation, auditability, and traceability on AI applications — regardless of where 
the AI algorithm is run. 


3. IBM has committed to release the largest publicly available dataset of over 1 million facial images to help solve 
one of the biggest issues in facial analysis—the lack of diverse data to train AI systems. 


To deal with possible sources of bias, IBM has several ongoing projects to test and address dataset bias in facial analysis 
— including not only gender and skin type, but also bias related to age groups, ethnicities, and factors such as pose, 
illumination, resolution, expression, and decoration. We are currently creating a million-scale dataset of face images 
annotated with attributes and identity, leveraging geo-tags from Flickr images to balance data from multiple countries 
and active learning tools to reduce sample selection bias. We intend to make this data publicly available as a tool for the 
research community and propose a challenge to encourage the community to improve their algorithms with respect to 
bias in facial analysis. The risk of bias being built into facial recognition AI systems is a concern for any organization 
developing facial analysis algorithms. Since AI is only as good as the data that trains it, IBM thinks making a diverse 
dataset like this available will help root out bias. 


4. We partner in multi-disciplinary and multi-stakeholder initiatives to collaborate and advance ethics in AI 


IBM is a founding partner of the Partnership on AI, a global initiative joined by more than 70 partners so far, that works 
hard to discuss and define best practices for beneficial AI in a collaborative multi-stakeholder environment. While 
executing research and articulating an evidence-based perspective as an industry leader is important, so is cross- 
industry collaboration. As such, IBM is a founding member of the Partnership on AI, a collaboration between many 
scientific and non-profit organizations. The organization’s main goal is to share best practices that encourage ethical 
development of AI, and to create a forum to discuss and resolve ethical and societal issues raised by the pervasive use of 
AI in our personal and professional lives. 


Reference Project Examples: 

In addition to IBM’s fundamental principles on AI Trust and Transparency and concrete examples of what IBM is doing 
to open the “black box” of AI, the following examples demonstrate how IBM is incorporating ethical frameworks, 
methods, guidelines and bias considerations in reference examples from M1: 
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to introduce bias into the analysis. 
thereby removing any potential bias. 


This foundational tenant — unbiased insight driven decisions 


are fully auditable, 
bias is not a factor 


The result 


can trust. 
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transparent, respect procedural fairness, and 
demonstrate high-quality data practices. This 


includes, but is not limited to, testing for 
outcomes and biases and fair, comprehensive 
and inclusive data collection 

practices. 


The supplier must provide examples of how it 
addresses ethical* practices when delivering AT. 
This includes demonstrating experience in 
applying frameworks, methods, guidelines or 
assessment tools to test datasets and outcomes. 
*For purposes of AI, ethical practices which are 
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PwC's Responsible AI Framework 


AI systems are finding their way into everyday life as they mature and gain trust, however, they also 
become susceptible to a range of new risks, including data privacy, exposure to data breaches, 
cyber-attacks, as well as data and algorithmic biases. Seventy-seven percent of CEOs in a 2017 survey 
conducted by PwC stated that they presume AI and automation will increase vulnerability and disruption 
to the way they do business. To mitigate these risks and maximize the potential AI benefits, leaders will 
soon have to consider applying a framework which provides trust and confidence and leads to adoption of 
the principles of a responsible AI (RAD, by making use of ethical, unbiased, transparent and 
secure models and workflows. 
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The PwC’s Responsible AI Framework Modules 


For concerns related to disruptive technologies like AI, the central theme is a need for responsible use and 
governance by an organization implementing AI, along with a need for greater regulations and oversight 
by government. PwC has a dedicated framework which provides a structure for the governance required 
for AI, as organizations begin to develop and implement various applications. 


Our RAI Framework helps our clients augment human intelligence with artificial intelligence 
technologies in a responsible manner. We validate that our clients’ entire modelling function from design 
to implementation and governance, including the gathering and preparation of data used in model 
training, testing and implementation, as well as the human-machine interface are responsible through an 
automated platform. Each phase of the RAI framework progresses a control structure that can be 
designed, implemented, and continually monitored to adhere to both business objectives, governance and 
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privacy requirements. Furthermore, we deploy distinct modules that address the fundamental aspects of 
responsible AI, as shown in the figure above. 


Pitfalls, Weaknesses and Critical Dependencies in 
AI Solutions 


AI has the same inherent risks and should be managed with the same discipline as any other technology 
enabled transformation. The largest pitfall is for organizations to buy into marketing hype and rush into 
developing an AI solution without having the five key foundations outlined below in place (in line with 
PwC’s Responsible AI framework). 


Clarity over AI strategy: 

e While there is a constant need to evaluate and adapt, it is still vital to be clear about the roadmap, 
with value-driving use cases constantly driving the path forward 

e Is the organization ready for the disruption from AI? 

e Has the organization considered the societal and ethical implications? 


e What business outcomes is the organization looking to achieve (e.g. product customization or back 
office efficiency)? 


Transparency by design: 
e Adoption of AI will be an emotive subject both within an organization, with their customers and in 
society as well, so it is important to consider how they will build stakeholder trust in the solution 


e Itis important to build the controls framework into the solution up-front rather than being designed 
and applied once systems are developed and in operation. This includes a mechanism to monitor 
outcomes and compliance 


Building the AI organization in advance: 
e There are many possible models for developing an organization-wide AI capability ranging from a 
centre of excellence and a dedicated board member to a ‘develop-and-steer’ strategy 


e Whichever approach is adopted, it is important to ensure cross-organizational communication, 
collaboration and centralized coordination of AI initiatives 


Data governance and management: 


e If data is the new IP, it is important to put in place mechanisms to source, cleanse and control key 
data inputs and ensure data and AI management are integrated. Poor data quality will lead to poor 
insights no matter how cutting edge the algorithm (or model) in place is 


AI operating model with Assurance Integration: 


e Assurance over AI is not a one-off. Risks and opportunities should be assessed as the AI platforms 
evolve 


e Assurance over AI also involves more than just embedding new technology into operational processes. 
It requires business-wide evaluation to gauge outcomes, identify emerging risks and look out for 
opportunities 


e Drawing on our wide-ranging research and work with clients, our Responsible AI framework is 
designed to provide transparency over the viability of the AI implementation project and confidence 
that the controls are in place to ensure that the business outcomes meet expectations 
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Prevention and Detection of Bias in AI/ML Models 


Referencing the Responsible AI Framework, a key step in the design of the solution is the Control 
Framework. The most effective controls are built during the design and implementation phase, enabling 
issues to be caught before they become a problem and also identify opportunities for improvement. An 
important question is who designs and monitors the controls? Both the breadth of application and the 
need to monitor outcomes requires engagement from across the organization. Control design requires 
significant input from business domain experts. Specialist safety engineering input is likely to be required 
for physical applications. 


A key part of implementation is breaking the controls down into layers (e.g., a ‘hierarchical approach’). At 
a minimum, there needs to be a hard control layer setting out ‘red lines’ and what to do if they are 
breached. Examples might include a maximum transaction value for a financial market trading algorithm. 
In more complex applications such as conversational agents, ‘behaviour inhibitors’ could be introduced 
that override the core algorithm when there is a risk of errors such as regulatory violation or inappropriate 
language. These core controls can be augmented by ‘challenger models’, which are used as a baseline to 
monitor the fitness and accuracy of the AI techniques or look for unwanted bias or deviations as the 
models learn from new data. Moreover, this approach can be integrated with continuous development to 
improve existing models or identify superior models for system upgrades. 


Responsible AI Use Cases 


We bring to life how PwC applies to projects its Responsible AI framework in the use cases below. Our 
framework is embedded in all of our AI projects to bring transparency and trust in the models that we 
deploy for our clients. 
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2.3 M3: AI Ethics 


2.3.1 Ethics & Corporate Responsibility in Thales Group 


As part of its ethical considerations, Thales has to take account of, and arbitrate between, sometimes 
contradictory stakeholder concerns. This leads to the signing of undertakings and charters that evolve out of a 
process of collaborative dialogue. 


Thales’s contributions to the International Forum on Business Ethical Conduct (IFBEC), the joint body of the 
AeroSpace and Defence Industries Association of Europe (ASD) and the Aerospace Industries Association of 
America (AIA) led to a draft proposal for a model Code of Conduct for suppliers, as well as an analysis of anti- 
corruption practices in terms of offsets, aimed at establishing global standards of integrity. 


Since 2003, Thales has been a member of the United Nations Global Compact. It defends the Global Compact's 
10 universal principles, and has integrated them into its strategy. In 2017, for the sixth consecutive year, thanks to 
high standards of reporting, Thales successfully submitted its Communication on Progress (COP) at the Global 
Compact Advanced level. Thales is the only French Aerospace and Defense Company to achieve this level of 
reporting. 


The NGO Transparency International ranks Thales in the top four European defense companies in its anti- 
corruption index (most recent ranking published in 2015). 


The Group also actively participates in French professional organizations (MEDEF, GIFAS, Cercle d’Ethique des 
Affaires (Business Ethics Circle), etc.) and international professional organizations (ASD (AeroSpace and 
Defence Industries Association of Europe), ICC (International Chamber of Commerce), B20 (international affairs 
community of 22 business organisations), IFBEC (International Forum on Business Ethical Conduct) dealing with 
business ethics, and has an active presence within the working groups of inter-governmental organizations 
(OECD, United Nations, etc.). 


The Code of Ethics, which is personally sent to each Thales employee, includes a foreword by the Chairman & 
CEO, and provides details of the integrity program. Thales created an Ethics & Corporate Responsibility 
Committee in 2001. The Ethics & Corporate Responsibility Department implements the policy defined by the 
committee. 


Thales scaled up its digital transformation in 2017, in particular through the acquisition of the big data specialist 
Guavus, the creation of a Digital Factory and the announcement of our plan to acquire Gemalto. The digital 
transformation of Thales and of governments and companies in general, calls for a radical review of the 
operational processes in place, and it raises new questions related to corporate responsibility and ethics. As part 
of Thales transformation plan, therefore, and as a matter of internal policy, Thales has taken steps to ensure that 
we specifically examine how these new digital technologies, and artificial intelligence in particular, might impact 
our business and our stakeholders. 


2.3.2 Ethical Practices when Delivering Al 


Today, the digital transformation and its related technologies - including artificial intelligence, Big Data, machine 
learning, ... - affect not only the relation of man to the machine, but also the relationships of individuals to each 
other. Becoming more and more powerful these technologies bring great advances in the daily life of the citizen, 
in the future of society and its environment but are also likely to create risks that must be mastered. 


Thales is committed to ethically-designed Al that is trusted and adopted by people — the human-centric 
development of AI. Whether for defence, transportation, aerospace or other applications, AI must enable users to 
connect to any network of objects and systems with trust and security. 
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2.3.3 Thales work on Responsible Al 


ical* practices when delivering Al. This includes 
, guidelines or assessment tools to test datasets and 


*For purposes of Al, ethical practices which are transparent, respect procedural fairness, and demonstrate high- 
Quality data practices. This includes, but is not limited to, testing for outcomes and biases and fair, comprehensive 
and inclusive data collection practices. 


They key to Al-powered human-machine interaction lies in the establishment of trust. 


° Al tech must understand and generate natural language. 

° The science comes first — an “explainable Al.” 

° There must be trusted communication links. 

° Must be able to clearly demonstrate how the machine works and learns. 
° Trusted Al must be inside the hardware. 


Responsible AI must integrate ethics into design, development and deployment in support of human-centric 
decision-making. As part of the trustable concern of Al solution and the need for traceability of the decisions 
proposed by Al systems, Thales has initiated research projects related to Explainable Al (XplAln™) to develop 
Trustable, Dependable, Ethical and Certifiable AI applications. 
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2.3.3.1 XplAIn™ definitions 


Generally speaking, non-interpretability of models can possibly be solved by applying and developing methods in 
the domain of Explainable Al (XplAln™). Both the motives for XplAln™ and the technical descriptions of 
explainable models are diverse and occasionally discordant. 


Oxford defines Explainability/Explain as: Make (an idea or situation) clear to someone by describing it in more 
detail or revealing relevant facts, Give a reason so as fo justify or excuse (an action or event), Be the cause of or 
motivating factor for. These definitions suggest that explainability refers to more than one concept even in plain 
English. 


In machine learning, XplAln™ refers to as artificial intelligence and machine learning techniques that can provide 
human understandable justification for their behavior. Zachary C. Lipton’ defines what explainability is and why it 
is important. Explainability is divided into trust, causality, transferability, informativeness, and fair and ethical 
decision-making categories and provides related work for each category (See Figure 2). 


It also investigates properties of interpretable models including transparency (e.g. simulatability, decomposability, 
algorithmic transparency), and post-hoc interpretability (e.g. text explanations, visualization, local explanations, 
explanation by example). Transparency refers to as the interpretability of algorithms and transferring them from 
black-box to glass-box whereas post-hoc interpretability justifies the decisions made by algorithms based on the 
resulted outputs. 


Figure 2 — XpIAln"" Taxonomy (Zachary) 


Algorithmic bias may be solved partly by selecting “precision” instead of “accuracy” as a metric for model 
selection. Another example is the use of methods to go from a black box approach (where the features used by 
the classification algorithm is unclear from a human perspective), to an interpretable approach, where the model 
is transparent, and decisions can be interpreted individually. The solutions target both the data, and the models 
themselves. The simulation will not attempt to reproduce a specific object detection and classification use case, 
but will rather reproduce the fundamental properties of such task. The goal is to have full control and knowledge 
on the “ground truth” in order to asses clearly what are the impacts of the tested solutions on our metrics. 


1 https://arxiv.org/pdf/1606.03490.pdf 
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PART IV. 
AI Ethics - Mandatory Criteria 3 (M3) 


When developing Al solutions, OVA follows the government of Canada guiding principles in terms of 
effective and ethical use of Al: 1) understand and measure the impact of using Al by developing and 
sharing tools and approaches; 2) be transparent about how and when we are using Al, starting with a 
clear user need and public benefit; 3) provide meaningful explanations about Al decision making, while 
also offering opportunities to review results and challenge these decisions; 4) be as open as we can by 
sharing source code, training data, and other relevant information, all while protecting persona! 
information, system integration, and national security and defence; 5) provide sufficient training so that 
government employees developing and using Al solutions have the responsible design, function, and 
implementation skills needed to make Al-based public services better 


As an SME, OVA has to protect its Intellectual Property (IP). Many of the tools that we develop are based 
on work done in collaboration with university partners. Given the nature of our collaborations, this 
means that some of our developments are published in part in scientific journals and presented at 
international conferences. This being said, details regarding proprietary algorithms developed are not 
communicated publicly in order to protect the company’s IP. We do our best to be as open as possible 
regarding our approaches and methodology, though not all is divulged. 


The solutions that OVA develops are for the most part centered on the user and its interactions with 
technology, whether it is a machine per se or an Al agent. Measurement of human performance is 
therefore paramount in many instances, especially when developing training and simulation platforms 
and intelligent platforms deployed in various industries centered on repair and maintenance. In such 
instances, users are always made aware when and how performance data is captured, and how the data 
will be used. Participants in our experiments are all assigned an alpha numeric code at the onset to 
make sure that data collected and the participant’s personal information are always decoupled. 
Informed consent is also always obtained prior to recording data from human participants during 
development and validation stages. 


Sharing the source code would not be a problem for OVA with the right legal clauses put in place to 
preserve its IP. In cases where the government is the sole client of a given solution, OVA would have no 
problem with code sharing. We also assist our clients at all stages of implementation of our solutions 
and proper training is provided. We currently offer a StellarX (OVA’s platform) Masterclass on how to 
use our platform at Laval University. 


Unlike many of our competitors, the solutions developed by OVA are developed with Al techniques that 
are for the most part explainable, especially when centered on decision-making. Regarding the latter, 
and depending on the project, we use human-in-the-loop or human-on-the-loop approaches. The 
output of our systems must therefore be understandable by humans since they are the ones in control 
in the end of the final outputs. Human performance measures developed with our tools are also easily 
correlated with actual actions performed by users. In many situations, it is very important that our 
algorithms provide actionable information to human operators. In the end, it is a human operator that 
will form a judgement based on the context and make the final decision at the end of the decision- 
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making process, so it is very important that the information provided by our tools be intelligible and 
useful. If results require expert knowledge to be interpreted, we failed to deliver the proper solution. In 
the end, our goal at OVA is to augment human performance. 


It is important to note that around 25% of our employees are women. An equal ratio of data collected 
from men and women are used during tools development to avoid any gender bias and women on our 
team participate actively at all stages of development. Participants in various age groups are always 
included in our analyses to make sure that models will also be age agnostic. In other words we apply the 
GBA+ standard proposed by the Canadian Government and we do our best to develop bias-free Al 
algorithms. 


In future AI projects we also propose to use the Algorithmic Impact Assessment (AIA) questionnaire 
designed to help you assess and mitigate the risks associated with deploying an automated decision 
system. The impact level of any automation proposed by us would also respect follow closely the 


proposed “Directive on Automated Decision-Making”. 


OVA is currently working on obtaining the Cybersecure certification from the GC. Given the nature of the 
work that we are currently involved in with the Department of National Defence, an audit of our data 
security management and infrastructure will be undertaken. We take very seriously to protect our 
clients (and participants) personal information. 


In conclusion, OVA is looking forward to working with the government of Canada and to use its knowledge 
in the field of Al and XR to implement innovative solutions. Our unique expertise - Al and XR - puts us ina 
great position to positively influence current methodologies used throughout the government. Our wide- 
ranging expertise we believe would benefit greatly the Government of Canada. 


Please accept our most distinguished salutations, 


Primary Contact: 
Harold Dumur (CEO) 
OVA inc 
hdumur@ova.ai 


Additional point of contact: 

Mathieu Boulanger (Chief Al Officer) 
OVA Inc 

mboulanger@ova.al 
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M3 — Ethical AI 


Ethical AI is at the core of Chillwall AI mission statement. In reference to the project outlined in 
this Letter of Interest, we will provide an outline of the Ethical AI we incorporated into our 
development. | | 


Chillwall’s ethical use of Al entails a two-pronged approach - impartial data generation 
processes and mitigation of algorithmic biases in model building. Our events data used to 
generate our modeling comes from a variety of open sources and we are proud to say that 
these events cover greater part of our world. Events data that we use is non personal and thus 
there are no questions or concerns that could be raised with regards to privacy. Even the data 
that we collect for creating personalized recommendations is optimally suited for our purpose. 
So, we just collect what we need. For example we use email ID to uniquely identify our users 
(their login ID), no demographic information is used to develop their personality models (used 
for coming up with recommendations). We are transactional focused. 


Our NLP based algorithms look at the usage patterns of words in the text, use unsupervised 
semantic similarity techniques, train supervised models using randomized sampling approaches 
and use balanced performance evaluation metrics. We envisage a reinforcement based 
approach to improve our recommendations that would solely rely on pinned event counts and 
clicking patterns, which in the absence of demographic data cannot be used to uniquely identify 
a certain individual, a major concern of privacy advocates. 


gA ChillwalL Al | Al Source List Letter of Interest (LOI) 
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Artificial Intelligence (AI) services, solutions and products | Executive summary 


AI is a tool that can be utilized in any industry, with an ever-expanding number of use 
cases, and which may leverage sensitive data. Given this breadth and depth, the 
consequences of misusing AI can be severe. This is why Deloitte developed and leverages a 
comprehensive AI ethics framework. 


Our framework focuses on ensuring that proper governance is applied to AI initiatives. It 
also highlights the importance of crafting thought-out approaches for ethical dilemmas 
and ethical breaches. Ethical dilemmas refers to the ‘grey’ areas of AI ethics, such as 
human-machine collaboration and the application of values to AI. Ethical breaches refers to 
the better understood aspects of AI ethics, such as privacy and transparency. 


We conclude our discussion of AI ethics by providing concrete examples of how we 
approach AI ethics with our clients. Please refer to section 4 for additional details on 
our AI ethics frameworks, practices, and examples. 
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Artificial Intelligence (AI) services, solutions and products | Our AI ethics practices [M3] 


4, Our AI ethics practices 


[M3] 


M3 


The supplier must provide examples of how it addresses ethical* practices when 
delivering AI. This includes demonstrating experience in applying frameworks, 
methods, guidelines or assessment tools to test datasets and outcomes. 


*For purposes of Al, ethical practices which are transparent, respect procedural 
fairness, and demonstrate high-quality data practices. This includes, but is not limited 


to, testing for outcomes and biases and fair, comprehensive and inclusive data 


collection practices. 


Requirement M3 and our response 


The supplier must provide examples 
of how it addresses ethical 
practices when delivering AI 


This includes demonstrating 
experience in applying frameworks, 
methods, guidelines or assessment 
tools to test datasets and 
outcomes. 


*For purposes of AI, ethical 
practices which are transparent, 
respect procedural fairness, and 
demonstrate high-quality data 
practices. This includes, but is not 
limited to, testing for outcomes and 
biases and fair, comprehensive and 
inclusive data collection practices 


what is strictly necessary 


63 


Section 4.4 of the response provides examples of how we 
work with clients to engrain ethical practices into the AI 
work we perform, and of how we use AI to drive just business 
outcomes. This demonstrates that we believe in both a just 
process and in driving just results. 


Section 4.2 provides the AI Ethics framework Deloitte follows, 
and section 4.3 expands on it and describes it in detail. The 
examples provided in section 4.4 identify which elements of the 
framework are addressed by each example. 


Both the examples provided in section 4.4 and the content 
provided in sections 4.2 and 4.3 address these topics. 
Specifically: 


e Section 4.3.3.4 addresses transparency, and section 
4.4 identifies examples that demonstrates 
transparency 

e Procedural fairness and high quality, comprehensive, 
and inclusive data practices are overarching topics 
discussed throughout section 4. For example: 

o Section 4.3.3.1 is on privacy and discusses how 
privacy must be considered from the onset of an 
AI tool, and data collection should be limited to 
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Artificial Intelligence (AI) services, solutions and products | Our AI ethics practices [M3] 


o Section 4.3.3.2 is on security and discusses the 
measures organizations can take to keep data 


safe 


o Section 4.3.3.3 is on data consent and discusses 
how users should be made aware what their data 
will be used for, and how permission is required 


before data is used for additional purposes 


o Section 4.3.3.4 is on transparency and discusses 


how users should be aware that AI is being 


leveraged and how organizations need to have a 


deep understanding of the AI tools they are 
using 


Deloitte applies a structured approach to ethics to ensure the Government’s AI solutions are 
governed with clear values and principles, follow the law, and are in accordance with human 
rights. Deloitte has a dedicated AI ethics practice to address these concerns and 
consistently apply AI ethic frameworks: 


e We have developed an AI Ethics framework to guide our thinking, and that of our 
clients. Our framework encompasses the usage of tools such as Privacy by Design 
and Algorithmic Impact Assessment 

e We have extensively applied our AI ethics framework in multiple engagements, with 
multiple clients, and throughout several industries and geographies 

e We have provided training on AI Ethics to some of Canada’s largest institutions 

e We have allowed Canadian institutions to gather valuable insights on their clients, 
and, as a consequence, offer them better services, without compromising the privacy 
and security of these customers 


In the following section we provide in-depth details that describe our AI ethics framework, 
approach, and provide examples of how we incorporate this thinking into our AI projects. 


4.1 Ethics and AI 


New technologies inevitably bring new opportunities and new challenges. The ethical 
implication of technologies and their implementation seem more prominent than ever before 
in our history. The pace of technology-driven change is so fast that policy-makers and 
regulators find themselves perpetually playing catch-up. Changes to policy, regulation and 
legislation take time. As a result, how can the government become proactive and not 
reactive on new technologies and their use? 


Despite the growing concern amongst the general population for data privacy, it is not 
uncommon for end users to accept complex terms and conditions, or provide access to their 
mobile devices’ information (e.g., contact list, location, photos), without properly 
understanding the terms they are abiding to and the potential implications of their actions. 
Details regarding data collection and use are often not available, or too complex from legal 
and/or technological perspectives for end users to have a clear understanding of how their 
data is being used. Conversely, 90% of Canadian respondents reported they would sever 
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ties with an organization if it was revealed that the company used data unethically.! This 
dichotomy presents a challenge to companies and government organizations to uphold 
ethical obligations, and as the ultimate custodian of the rights of Canadians, it is crucial for 
the Government of Canada to play a pivotal role in the development of ethical standards 
and the setting of ethical precedents in AI. Canada’s global reputation and commitment to 
transparency, inclusion and collaboration presents Canada with an opportunity to lead the 
world on AI ethics practices. 


Ethics is a multifaceted topic that requires stakeholders to have a clear understanding of the 
power of AI and its applications. This raises several ethical considerations for the 
Government of Canada to consider as it forms its perspective on how to leverage AI. For 
example: 


e Does the original intended use of the AI solution justify the potential risks and the 
impact it may have on the public? 

e Are there societal concerns that need to be addressed, including, but not limited to 
privacy or legal? 

e Are there proven assessment methods in place to detect bias and ensure fairness? For 
example, detect data that present bias against, or in favour of, a particular segment 
of society 

e Do we have a clear and transparent method to scrutinize, interpret and explain the 
algorithm and its results? 


This is a sample of factors to consider for any AI application being evaluated by a public entity. 


4.2 Deloitte’s principles and considerations for AI Ethics 


Deloitte firmly advocates for the consideration of ethics principles in the implementation of 
AI, and we embed ethics when building and deploying AI solutions. 


1 Deloitte, “Privacy for sale — to the highest bidder”. Accessed at: https://www2.deloitte.com/content/dam/Deloitte/ca/Documents/deloitte-analytics/ca-en-data- 
ethics-and-privacy-survey.pdf on October 8", 2018. 
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Artificial Intelligence (AI) services, solutions and products | Our AI ethics practices [M3] 


Figure 6: Deloitte's Ethics considerations have 


Further details on each of these pillars of AI ethics are provided below, along with 
frameworks and examples of how Deloitte applies these principles in practice. 


4.3 Applying Deloitte’s Ethics Frameworks, Principles and Guidelines 
when delivering AI 


4.3.1. AI 


It is important to recognize that the Government plays multiple roles in the AI ecosystem. 
First, the Government creates the policies, regulations and laws that protect Canadian 
individuals and institutions from uses of AI that contradict Canadian values. Second, the 
Government leads by example as it adopts AI following well-crafted ethics guidelines. In 
both cases, ethics 

must be developed to ensure policies and guidelines are available to protect the 
general public’s interest. Setting the right , for example 

, is integral to future AI applications. 

Beyond the definition of the processes themselves, training government employees and 
partners will be important to ensure that all stakeholders are familiar with ethics 
expectations and aware of risks and considerations. 


Deloitte has developed an AI Ethics to help 


organizations guide and structure staff to address existing and emerging risks across the 
lifecycle of an AI deployment. The should consist of 
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that take into the ethical considerations of AI, and 
provide on new and changing initiatives. 


An example of how Deloitte leveraged this framework to accelerate AI through 
is provided on Section 4,4 - Example 8. 


It is essential that the Government monitors and adjusts their AI solutions by following a 
standardized , no different at a conceptual level from that used 
for any other technology program. A sample AI , along with its 
key elements, is described below. 
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Figure &: Sample Al 
This should be embedded into the Government’s IT governance process and triggered 


any time a change is planned for an AI-based solution. 
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is part of Deloitte’s approach to AI 
Throughout a Deloitte helps an organization build an effective 
in support of the use of AI. This contributes towards the organization's ability to 
considering the 


requirements to effectively manage personal information risk. 


Figure O: Deloitte's Frameworio, typically leveraged as a component of Al 


Providing © to relevant actors involved in all aspects 
of the AI solution is a key component of a successful AI implementation. Deloitte’s 
helps establish impactful by identifying the organization’s 
needs and delivering a We believe the goal of an effective AI 
is to 


as well as the organization’s collective ability to approach the topic. This is 
accomplished through a unique understanding of: 
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An understanding of this intersection will position the Government of Canada to consistently 


build the AI solution while providing a means by which 
in ongoing of key actors. Deloitte 
implements this we provide to our clients, while also providing 
that can last after the project’s 
completion. Deloitte’s supports the following key program 
elements. 
Figure 10: Deloitte's Key Glements 


Another approach to raise awareness of AI ethic considerations is 
AI may be unfamiliar to many relevant 
actors and can be overwhelming, therefore 
is needed to in a way that is transparent and relevant. Deloitte’s 
are an example of Deloitte’s approach to 
An example of AI ethics considerations 
is provided in (Section 4.4 - 


Example 6). 
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4.3.2. 


In this section we discuss two major aspects of 


AI 


Creating sophisticated, and in some cases autonomous, decision systems inherently brings 
with it two challenges: 
Before implementing an AI 
application organizations must consider in the first place if the application 
and if so, 


Assessing use cases for AI 


We advocate that organizations seek to develop AI applications that have a positive 
outcome for people and society, and that minimize the potential to cause harm. To achieve 
that, the first decision an organization needs to make is 

The organization should consider questions such as: 


The Government must be able to assess AI use cases and draw a line that best represents 
not just what Canadian law states, but the beliefs and values of Canadians. This is not just a 
responsible approach, it is also key in obtaining citizen acceptance of AI. 


Embedding our values 


To increase the likelihood of deploying AI in an ethical manner, the Government should 
consider how to develop AI applications that promote Canadian values. For example: 


© Deloitte Inc. and affiliated entities 
7i Solicitation No.: EN578-180001/A 
Deloitte Proposal Number: 4816 


Divulgué en vertu de la LAI 
Released under the ATIA 


000289 


s.20(1)(b) 


s.20(1)(c) 
Artificial Intelligence (AI) services, solutions and products | Our AI ethics practices [M3] 
Fundamentally, it is critical that 
An example of how we worked with a top-tier Canadian 
but to use AI to is provided in (Section 4.4 - Example 7). 
It is also important to recognize that discussions on topics such as are 


subjective. For example, we need to ensure that any AI deployment has a 
across the population in question. However: 


Answers to questions such as these are not yet available. The government will need to work 
closely with advisors and with Canadians in order to 

It is vital therefore that the government select AI partners who emphasize the 
importance of 


in practice: credit card adjudication 


A common AI application is credit card adjudication. If we utilize a company’s dataset for 
the last 20 years to predict whether a new customer will have the ability to pay back their 
credit card debt, we have to be sure that we have a very accurate and representative 
sample for today’s society. For example, a credit card company who has been in existence 
for over 20 years will need to be very confident that how they approved and declined 
customers 20 years ago is still relevant today. Usually this is not the case, as times and 
cultures change and evolve. Therefore, we must 


How Deloitte approaches 


At Deloitte, we utilize various tools 
To date, we've conducted: 
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When we speak with clients, we often tell them that AI is changing the way businesses 
perform their day-to-day operations. 


This is an important step in the process as it provides a direct 
channel for all AI inquiries and issues, 


Furthermore, we advise clients to 


It is important that we all keep an open mind when deploying AI, so that we 
strengthen the ability for AI to help us for the good and betterment of society. 


For concrete examples of how Deloitte has addressed matters of in 
client projects, refer to section 4.4. 


Muman-machine Collaboration and Coexistence 


Even if AI applications are intended for the benefit of people and society, and have a strong 

, there are still concerns regarding human-machine interaction. 
Research shows that between 35% and 42% of jobs in Canada are highly susceptible to 
automation”. We believe that while AI is poised to be disruptive, this disruption is not 
necessarily negative. In the short-term, AI has the potential to free up employees from dull, 
dangerous, and/or repetitive tasks, putting a focus on higher value work. As such, it is 
necessary to explore and understand the ways in which humans and machines can work 
together. An example of how we worked with a large company to embed AI into 
their operations in a way that leveraged the best of their people and resources is 
provided in (Section 4.4 - Example 8). 


The two main models of co-existence: semi-autonomous and fully autonomous 
When looking at human-machine collaboration, we can usually distill the conversation to 
two options. We either want a human in the loop (semi-autonomous), or no human in the 
loop (fully autonomous). There can be arguments made for both models, and it really 
depends on the application to determine which method is best. 


For semi-autonomous collaboration, a human will always have the final decision-making 
authority. This ensures that any fairness or integrity issues in the AI application are vetted 
by a human, who presumably is an expert or domain specialist. This strengthens the ability 
to ensure that AI applications are fair and accountable. On the other hand, fully autonomous 


? Deloitte, “The Intelligence Revolution Future-proofing Canada’s workforce”. Accessed at: 
https://www2.deloitte.com/content/dam/Deloitte/ca/Documents/human-capital/ca-EN-HC-The-Intelligence-Revolution-FINAL-AODA.pdf on October 8th, 2018. 
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machines will make decisions and run without any human intervention. There could be 
situations, such as large volume deployments, where human intervention is neither practical 
nor useful. In these situations, we must 


Collaboration and co-existence in practice: weaponized AI 

Weaponized AI serves as a useful case study of contrasting approaches to the semi- 
autonomous vs. fully autonomous consideration. In the United States, the government has 
indicated that their weaponized AI should always be semi-autonomous, with a human 
operator always making the final decision to strike. On the other hand, China has stated 
that they would like their weaponized AI to be fully autonomous. This is a striking example 
of how challenging it can be to set global regulations and standards, when different 
countries often have different cultural values and norms. 


An example of how we worked with a top-tier Canadian 
to ensure bias was not present in an AI application is provided in (Section 4.4 - Example 1). 


Our perspective on collaboration and co-existence for organizations 


One of the key ways to 


For example, 
organizations can focus on: 


For example, a Deloitte study found that, with strong support for 
adoption of automation and AI applications, US states could expect 27 to 30 percent time 
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savings for a given use case. Similar results can be expected in Canada, Realizing this 
benefit will require investments in technology as well as rethinking many aspects of how 
government works. 


A second aspect of co-existence has less to do with organizational, and more to do with 
individual, behaviour. Automation bias is the notion that humans will trust machines even 
more so than fellow humans. We have often heard stories where humans will follow GPS 
navigation devices into a swamp. There are countless examples where humans think 
machines and algorithms are smarter than our own common sense. 


To do this, Deloitte often encourages clients to 


4.3.3. Ethical Breaches 


Within Ethical Breaches we consider four principles that help organizations mitigate the risks 
and impacts of a potential breach in sensitive information. These principles are: privacy, 
security, data consent, transparency. 


Privacy 
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Privacy by Design is an internationally recognized framework based on the premise that 
privacy should be proactively embedded into the design, operation and management of 
IT systems, networked infrastructure and business practices. Organizations can get their 
privacy practices verified by getting certified in this framework, thanks to a partnership 
between Deloitte and Ryerson University. Ryerson handles applications and issues 
certificates, while Deloitte’s privacy and information security professionals conduct the 
initial assessment. Far from a tick-the-box compliance exercise, this involves testing the 
organization’s privacy controls against a well-defined set of measurable criteria and 
illustrative controls, developed by Deloitte professionals. Deloitte also assesses the 
strength of one’s practices against internationally recognized privacy principles. 


The Privacy by Design framework is based on 7 foundational principles: 


1. Be preventative, not remedial. Anticipate and prevent invasive events before they 
happen, not scramble to manage after a breach. 

2. Lead with privacy as the default setting. Ensure personal data is automatically 
protected; don’t require users to take extra steps to do so. 

3. Embed privacy into the design. Privacy measures should be fully integrated 
components, not added on later. 

4. Retain full functionality. Privacy and security are equally important; neither should 
be compromised for the other. 

5. Ensure end-to-end security. All data should be securely held while it’s needed and 
destroyed when it’s not. 

6. Maintain visibility and transparency. Assure stakeholders that business practices 
and technologies involved are transparent to the end-user and subject to 
independent verification. Remember: it’s not your data. 

7. Respect user privacy. Individual interests must be supported by strong privacy 
defaults, appropriate notice and user-friendly options. 


Data Minimization 


In Canada, the Privacy Act requires minimizing the collection of PI. This can have an impact 
on the development of AI solutions due to the reduction of data being used within the 
model. Deloitte recognizes the privacy implications of big data and over-collection of 
sensitive data and address these issues. 


Access, Correction and Destruction of PI 


Federal entities are required to facilitate access requests made under the Privacy Act and 
the Access to Information (ATI) Act, which becomes more complex when incorporating AI 
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with the right of individuals to access their data. 


Deloitte also understands the increasing trend of adding the ‘right to be forgotten’ to access 
rights of individuals, arising from legal instruments such as the European Union’s General 
Data Protection Regulation (GDPR) and possible amendments to Canada's Personal 
Information Protection and Electronic Documents Act (PIPEDA). Although this right is not 
yet in effect in Canada, the government may want to be at the forefront in meeting this 
increasingly present expectation. 


Security 


Although the use of emerging technologies provides great opportunities to streamline 
processes and create greater organizational efficiencies, it can also expose the organization 
to greater security and privacy risks, while increasing reporting obligations in instances of 
security breaches. 


Currently, there are different industry options and methodologies for security control 
assessments, such as NIST, CIS/SANS 20 and ISO 27001. 
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Figure ti: Deloitte's Model 
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Figure 12: Deloltte's framework 


Bata Consent 


As AI applications, regulations, and end user expectations evolve, it is vital that data 
consent principles and standards be defined, applied, enforced, and understood by the 
general population. 
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Transparency 


4.4 Examples of how we have applied our ethics framework in our work 


In this section, we present examples of how we have worked with our clients to embed 
some of the various concepts and methodologies described above, ensuring that some of 
the biggest organizations in Canada, and around the world, make ethics an integral part of 
their AI applications, from ideation to post-implementation maintenance. Below is a 
summary of how each example aligns to our AI ethics principles. 


Figure 13: summary of how we have applied Al ethics principles in our client projects 
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Context 
e Deloitte was engaged by a 
major Canadian to 


Ethics challenge 


Context 
e A major Canadian 
engaged Deloitte to create 


Ethics challenge 
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Context 
e One of the biggest providers 
in 
Canada approached Deloitte 


Ethics challenge 


Context 
e The client, 


was 
facing a potential lawsuit 


Ethics opportunity 
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Context 

e A major Canadian 
acknowledging the 
importance of | 


Ethics Challenge 


Context 
e A major Canadian 
engaged Deloitte 


Ethics opportunity 
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Context 
e A major Canadian 
identified 


Ethics opportunity 


Context 

e Alarge company 
was undergoing business 
model transformation. 
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Ethics opportunity 
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M3 - ETHICAL PRACTICE IN AI AT SIMON FRASER UNIVERSITY 


Simon Fraser University 1s committed to acting ethically in all aspects of our work. This 
includes: 


Statement: Ethical Conduct for Research Involving Humans (TCPS II) 

Setting high standards for academic integrity, including respect for knowledge, truth, 
scholarship and acting with honesty 

Actively engaging in reconciliation and decolonizing work through working to 
implement the Truth and Reconciliation Commission’s Calls to Action 
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Serving as an institutional leader in pursuing ecological, social, and economic 
sustainability 


We approach our work in AI with the same high standards for ethical practice. Our expert team 
members include researchers who are raising the bar for what it means for AI work to be ethical. 
For example: 


Dr. Peter Chow-White heads up the GeNA Lab, seeking to understand the opportunities 
and risks of big data technologies by engaging key stakeholders, who develop and use 
them, including scientists, entrepreneurs, users, and the public. 

Dr. Fred Popowich (see bio above) serves on the SFU Research Ethics Board, bringing 
his expertise to the ethics review of AI- and Big Data-related research from across SFU. 


SFU is the only Canadian university participating in the initiatives of AI¢ALL, a US-based non- 
profit dedicated to increasing diversity and inclusion in AI education, research, development, and 
policy. We recognize that diverse perspectives are a critical factor in preventing bias in Al and in 
ensuring that AI systems reflect the values and needs of the people they are intended to serve. 


The specific tools and methods our experts use vary by project. The general principles we follow 
include: 


In collaboration with clients, clarify the specific problems to be solved and the types of 
data that are necessary (and those that, by extension are not necessary) for addressing 
those problems. 

When developing data collection methods, work with the client to identify the most 
relevant sources and means of obtaining necessary data to create fair, comprehensive, and 
inclusive datasets, and help the client to understand limitations inherent in the resulting 
datasets. 

When using client-supplied datasets, seek to ensure the client understands the specific 
data needs and can provide us with targeted data, mitigating the privacy issues inherent in 
using opportunistic datasets that may be much broader in scope than is needed to address 
the problem. 

Test datasets for internal consistency and for issues such as sparsity or biases, 
recognizing that understanding problems inherent in a dataset enables our experts to 
adopt or develop ways to mitigate the impacts of imperfect data. 

Train models on the appropriate datasets and test the outcomes for biases, such as over- 
or under-representation of specific categories. 

If biases are detected in outcomes testing, address them through additional training data 
or refinements to the model itself. 
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The supplier must provide examples of how it KPMG is a thought leader on 
addresses ethical* practices when delivering Al. This the impacts of digital and how 
includes demonstrating experience in applying advanced technologies need 
frameworks, methods, guidelines or assessment tools to address a range of societal 
to test datasets and outcomes. factors including ethics. 


*For purposes of Al, ethical practices which are Refer to page 33 fora 
transparent, respect procedural fairness, and description of our frameworks, 
demonstrate high-quality data practices. This includes, | methods and approaches. 

but is not limited to, testing for outcomes and biases 

and fair, comprehensive and inclusive data collection 

practices. 
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Many businesses are currently developing and operationalizing Robotic Process Automation (RPA) solutions and are 
exploring true Artificial Intelligence (Al). These are systems that can both interpret natural language and also learn to find the 
right answers at the speed of light without them having been pre-programmed. The rapid development and evolution of 
Artificial Intelligence, while unleashing opportunities for business and communities across the world, have provoked a number 
of important questions relating to ethics that go beyond the walls of academia and high-tech research centres. Governments, 
business and citizens alike are demanding more accountability in where and how Al technologies are used, trying to find 
solutions to the legal and ethical issues that accompany the growing integration of Al in human activity. In business ethics in 
particular, the rise of artificial intelligence has brought about considerations including the level of control organizations can 
retain over their machines’ decision-making processes and how to ensure that the Al systems they adopt always act in a way 
that is in line with the organization’s core values. Al has become increasingly powerful and its applications to business are only 
starting to reveal its potential — with great power comes great responsibility. 


SR 


Globally, KPMG has been at the forefront in actively participating to draft legislation and standards to address the legal and 
ethical issues associated with AI. In Canada, KPMG's Sylvia Kingsmill, Partner Canadian Privacy, Regulatory & Information 
Management Leader, Risk Consulting is leading the way in her roles as: 


— Advisory Lead, CIO Strategy Council, AI & Ethics Standard 
— Chair, ISO PC/317 Consumer Protection: Privacy by Design for Consumer Goods and Services 


— Special Advisor to the International Council on Global Privacy and Security by Design, advising on the deployment of 
privacy and security enhancing technologies involving artificial intelligence, big data, robotics and machine learning 
initiatives. 

— Creator, Advisory Lead for the Privacy by Design Certification Program, launched in partnership with Dr. Ann Cavoukian for 


the new General Data Protection Regulation (GDPR). 


— Interim Chief Privacy Officer for a Canadian non-for-profit organization with global health information management 
operations. 


We believe it is important to first look at what legislation and standards exist that would inform ethical practices when 
delivering Al. 
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With the exception of the General Data Protection Regulation (GDPR) in the EU which came into force on May 25" that 
regulates automated decision making, there are no consistent cross-industry standards or laws that currently exist to govern 
the development and use of Al and machine learning. The United Sates (US) has recently passed a bill on Al giving the NYC 
government two years to create a task force that provides recommendations on how information on agency automated 
decision system may be shared with the public and how harms and algorithmic bias will be addressed. 


Despite the lack of a uniform industry standard or legal approach to regulating Al, the international community comprised of 
national governments, research bodies, regulators, and consumer advocacy groups have advocated for the creation of 
framework and guiding principles for Al hat provides a more holistic approach to safeguarding the privacy rights of individuals 
than just ensuring the confidentiality of their data. 


The discussion is centered on the need for growing awareness about the societal, ethical and legal implications of Al anda 
principled approach to algorithmic risk management that goes beyond the law calling for strong accountability measures, 
ethics, data stewardship, policy oversight and transparency to avoid the pitfalls in Al and Machine Learning becoming a “black 
box’—which often results in biased, unfair and discriminatory decision making. 


There are several challenges relating to Al that need to be addressed to foster trust and adoption of such advanced 
technologies, including: 
— Accountability for the use of data 


— Ethics 

— Consent 

— Purpose limitation & Data minimisation (versus Accuracy) 
— Bias, Fairness and Transparency 


— Right to access and challenge information. 


Each of these principles are interrelated and form the basis of KPMG’s Al Risk Framework which we outline later in this 
section. 


The European Union's (EU) Approach to Al — the GDPR 


The GDPR specifically contains “the right not to be subject to a decision based solely on automated processing” and 

mandates the lawful, fair and transparent processing of data, which applies directly to Al. It was designed to address the risks 
of unfair and biased decision making about EU data subjects by providing them with a right to opt-out of certain profiling and/or 
automated decision making, to obtain a meaningful explanation into the algorithmic decision making process and to challenge 
the output. This means providing individuals with the ability to ask how and on what data their algorithm is trained. 


The GDPR places heavy emphasis on accountability, which is the foundational principle articulated under Article 5, mandating 
fair and lawful processing, including with respect to profiling and automated decision making. Two requirements especially 
relevant to the use of Al under the GDPR are Article 25 — Privacy by Design and by Default; and Article 35 — Data Protection 
Impact Assessments, both of which are detailed further below. Canadian organizations will soon face regulatory reform in 
Canada, being currently urged both by our federal government and the Privacy Commissioner, which in effect will likely require 
a similar level of protection to that of the GDPR for the processing of data. Of note is the federal report issued in February that 
put forth 19 recommendations to amend the private sector law, the Personal Information Protection and Electronic Documents 
Act (PIPEDA) calling for algorithmic transparency requiring companies to include privacy, security and ethical standards in the 
process of developing an algorithm, right from the design to the testing and launch stages. 


As such, it is prudent to look to the EU’s GDPR requirements for the use of Al technologies for a baseline of privacy 
protections and specifically, to the principles mentioned above. 
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Privacy by design means building privacy protections into software, new systems and solutions by leveraging privacy by 
design thinking, ensuring that strong protections are in place by default, appointing a Data Protection Officer to ensure 
responsible use of data and demonstrate accountability through data protection impact assessments that measure and 
address “high risk processing” and by implementing organizational and technical safeguards when organizations profile and/or 
automatically process data. Data minimization and purpose limitation go hand-in-hand to ensure that organizations only use 
the data they need and for a specific, identified purpose in line with what an individual would expect or consent to. Individuals 
should also be granted the right to access, correct and challenge the decisions made about them via Al systems, which goes 
to the principle of transparency, especially in reference to the data inputs given that bias can be written into the code of the Al 
and if poorly designed, will include the bias of the developer. 


Data Protection Impact Assessments are important tools that outline the methods of data collection (algorithmic inputs), the 
processing itself throughout the entire data life cycle (which should also include the lifecycle of the algorithm itself, from data 
selection to the design and integration and live use case as it goes into production). It is used to assess how the 
implementation of new technologies impacts privacy and qualifies the risk levels based on impact and likelihood with a 
prioritized roadmap or report of findings, including recommendations, for reducing the privacy harm or risk. 


Industry approaches to preserving confidentiality of the data which can in effect be followed to comply with Article 32 (Security 
for Processing) include the use of encryption, such as “homomorphic encryption” that enables data processing to take place 
while still in encrypted state, which in effect, prevents unauthorized access, disclosure or compromise/breach of data. Other 
state of the art technical controls under Article 32 mention anonymization or privileged access controls for example, where 
only authorized users may access data and for a specific purpose, in line with the expectations of the individual who provided 
consent for the collection and use of their data (unless another legitimate ground for processing data can be relied on). 


The Norwegian Data Protection Authority has recently published a report on “Al and Privacy” summarizing their 
recommendations to industry (developers, system suppliers), end users, legislators and political decision makers for protecting 
personal data when developing and using Al systems, building on the GDPR, including: right to information, consent, right of 
access, right to rectify/object/limit processing, conducting risk assessments, complying with privacy by design, limiting data 
collection and use, as well as regular testing — all of which is consistent with the foregoing analysis contained below and 
above. Of note was the continuing need to establish industry norms, ethical guidelines and a data protection panel consisting 
of external experts that can advise on the challenges and opportunities. 


Canada's Approach to Artificial Intelligence and Machine Learning (AIML) 


It is important to highlight some of the emerging Canadian work in the field of Al and privacy by The Information Accountability 
Foundation (IAF) and the Office of the Privacy Commissioner of Canada recently completed a Canadian research project that 
resulted in a Canadian Assessment Framework to help assess big data analytics and data processing initiatives to determine 
whether they are legal, fair and just. This is to complement the Canadian private sector privacy law, namely The Personal 
Information Protection and Electronic Documents Act (‘PIPEDA”), which requires organizations to be both accountable and to 
obtain consent to process data, but is silent on the ethical constructs required for ethical data processing. 


In short, PIPEDA requires accountability by requiring organizations be responsible for the personal information under their 
control and to protect data using “adequate” and “reasonable” security safeguards to protect the confidentiality of individuals, 
yet it does not prescribe any mechanisms or controls for doing so for emerging technologies. Similarly, there are limitations in 
our public sector law, the Privacy Act, which requires consent or legal authority to collect personal data, but is silent on 
principles such as fairness and accountability. 


As such, this Canadian Assessment Framework developed by the IAF layers in values and ethical considerations that go 
beyond legal requirements and provides guidance on how to be transparent with the data without publishing proprietary 
materials about the algorithm itself. It means Canadian organizations will need to go beyond the law to foster privacy best 
practices in developing processes for the oversight, monitoring, and accountability of Al systems to ensure their integrity and 
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trust, and more importantly, in ensuring a process is in place for individuals to challenge any seemingly biased or 
discriminatory decisions made about them. 


The IAF has also established and Al and Ethics Project to address ethical data processing by suggesting a governance 


structure based on 5 “Essential Elements of Accountability” which have recently been updated (in 2017) to include Al Essential 


Elements, summarized below as follows: 

— Organizational commitment translated into policies and processes that outline values 

— Mechanisms to translate core values into privacy policies 

— Internal ongoing oversight via data protection impact assessments 

— Transparency mechanisms where decisions are communicated widely 

— Remediation and external enforcement to demonstrate soundness of internal processes. 


Another leading Canadian Framework for Al and ethics is the pioneering work of Dr. Ann Cavoukian, the former privacy 
regulator for the province of Ontario, who has developed the concept of privacy by design, now an international privacy 
standard since it was adopted unanimously in 2010 by global Data Protection Authorities and also law under Article 25 of the 
GDPR. She has developed the following Al Ethics Framework based on the foundational philosophy of building smart default 
settings into architectures and business practices from the start to proactively manage risk: 


— Transparency and accountability of algorithms essential; 

— Ethical principles applied to the treatment of personal data; 

— Algorithmic oversight and responsibility must be assured; 

— Respect for privacy as a fundamental right; 

— Data protection/personal control via privacy as the default; 

— Proactively identify the security risks, thereby minimizing the harms; 
— Strong documentation to facilitate ethical design and data symmetry. 


Finally, there is an important body of work emerging out of the Institute of Electrical and Electronics Engineers (IEEE) that 
recently launched a global initiative to create standards for autonomous and intelligent systems that will prioritize the ethical 
and well-being for all aspects of Al. These ethical standards will serve to build upon and complement the privacy principles 
and best practices frameworks established in current Canadian law, Dr. Cavoukian and the IAF. 


Our Risk Management Approach to Ethics in the use of Al 


At KPMG we believe that for Al to be truly transformative we must have confidence in how it functions. That's why a 


comprehensive assurance model for Al is so important - one that builds trust through guaranteeing that algorithms are reliable, 


the system is cyber secure, IT processes and controls are properly implemented, appropriate data management is in place 
and that there is a governance structure that understands the ethics of machine learning. That understanding is subsequently 
included in the management of a wide range of organizational risks, like the potential impact of failure on financial results or 
reputation. 


In order to harness the societal benefits of Al and reap the operational, financial and strategic decisions these emerging 
technologies will bring, we must also consider, understand and proactively manage the risks. This is because academic 
research, the media and regulators have all revealed that the algorithms behind e-commerce sites, dating apps, search 
engines and social media sites, to name a few, have resulted in bias, class discrimination, injustice, racism and/or sexism. 
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In the US, it is not uncommon to see courts leveraging Al to decide when and for how long criminals should be jailed—one 
such system currently being used is the Public Safety Assessment in Texas and New Jersey. It was designed to provide 
judges with the most objective information to make fair decisions about prisoners by reducing biased rulings that can be 
influenced by a defendant’s race, gender or appearance—because the risk factors used do not include race, gender, 


employment background, where a person lives or history of arrests that can create biased decisions. What is paramount to its 


success and keeping dangerous offenders off the streets is the transparency in how the risk score was derived and the 
requirement of judges to be involved in the final sentencing decision. 


In fact, the entire policy debate around the use of algorithms in Al has been focused on the need for transparency, fairness, 


governance, and equity to correct for the digital biases that may result if left unchecked without human intervention. The 


Federal Trade Commission asserted that organizations must correct for biases inherent in automated processes at both the 


collection and analytic stages, and balance the predictive value of the model with fairness considerations. 


The common algorithmic pitfalls/risks include the following: 

— Inaccurate, incomplete or irrelevant input data 

— Human bias in training, coding, logic or inappropriate modeling resulting in flawed algorithmic design 
— Inappropriate data collection (lacking consent or legitimate basis) 

— Inappropriate contextual use or assumptions about the data outputs 

— Lack of implementation or integration with business operations resulting in inappropriate decisions 
— Lack of governance and/or integration with organizations values and principles 

— In appropriate or incorrect use of the output data resulting in bias, discrimination and/or unfairness 

— Technical design flaws due to lack of testing, validation or training of algorithm 


— Security and privacy flaws due to intentional/malicious or hacked data inputs which result in deliberate and flawed 
outcomes 


— Lack of transparency given the proprietary nature of the solution resulting in decisions being made in a “black box”. 


In addition to the above-noted risks, there are organization-wide implications, including: 


— Strategic and financial risks: Errors can put an organization at public risk when algorithms are used to rely on making 


financial and/or strategic decisions resulting in financial loss or in accurate reporting due to lack of integrity 


— Reputational risks: If values, principles and ethics of an organization are not aligned with algorithmic outputs which 


may be used to manipulate stakeholders, citizens, regulators or politicians, not to mention media scrutiny and negative 


publicity 


— Regulatory risks: Flawed algorithmic decisions can be used to discriminate and/or violate rule of law, constitutional 


right and freedoms, including privacy rights, exposing organizations to lawsuits or Charter challenges or judicial reviews 


and/or regulatory investigations. 


The recent Declaration on Ethics and Data Protection in Artificial Intelligence’ identifies the following ethical practices 
(summary), which inform our Al Risk Management Framework outlined in the next section: 
1. Artificial intelligence and machine learning technologies should be designed, developed and used in respect of 
fundamental human rights and in accordance with the fairness principle 


1 https /Hedpac.or ioads/201 8/10/2071 80922 ICDPPC-AON Al-Declaralion ADOPTED pdf 
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2. Continued attention and vigilance, as well as accountability, for the potential effects and consequences of, 
artificial intelligence systems should be ensured 


3. Artificial intelligence systems transparency and intelligibility should be improved, with the objective of effective 
implementation 


4. As part of an overall “ethics by design” approach, artificial intelligence systems should be designed and developed 
responsibly, by applying the principles of privacy by default and privacy by design 


5. Unlawful biases or discriminations that may result from the use of data in artificial intelligence should be reduced 
and mitigated 
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Example of how KPMG addressed Al Ethical Practices 


A large international client leveraging KPMG to deliver an Al solution was concerned about the future potential of decisions 
being made by the Al solution. This included potential misalignment to the organization’s cultural and ethical values, or the 
potential to cause bad or incorrect decisions by a human employee / Al solution, resulting in the organization being held 
accountable. Poor quality or incomplete data made available to the Al solution will impact the quality of the decision taken in 
terms of compliance with corporate values. Without periodic reviews of the Al logic and the data used by the Al solution, it 


could deviate from the organization’s corporate values. 


To address these concerns, where possible, ethical rules and corporate values were coded into the algorithms, and controls 
were put in place to review the output (e.g. thresholds; list of acceptable outcomes; list of unacceptable outcomes). Changes 
to the ethical value code go through robust change management process. The controls deployed match the velocity and 
breadth of the process being controlled. During the design phase, a multidisciplinary team brainstormed about the potential 
ways the Al solution or its outcomes could be misused, e.g. the Al solution or the company itself falls in to the ‘wrong hands’. 
Data was reviewed to ensure it was complete, accurate and free from bias. Logic was reviewed and tested to verify that it 
remains valid, with specific testing to ensure there are no un-intended biases. KPMG’s governance framework leveraged the 


COBIT process, MEA02 Monitor, Evaluate and Assess the System of Internal Control. 
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s.19(1) 


s.20(1)(c) 


. M3 . The supplier must provide examples of how it addresses | See below. v See below. 
- | ethical* practices when delivering Al. This includes - - - 

. demonstrating experience in applying frameworks, 

. methods, guidelines or assessment tools to test datasets 

and outcomes. 

| *For purposes of Al, ethical practices which are | 

| transparent, respect procedural fairness, and demonstrate. 

high-quality data practices. This includes, but is not limited — 

_ to, testing for outcomes and biases and fair, 

. comprehensive and inclusive data collection practices. 


ServiceNow Response: 


ServiceNow provides an ethical and thoughtful approach to Alin many ways. This includes the transparent Al products, the disciplined 
practices applied to data management and the thought leadership in the Al soace (detailed below). As a company, ServiceNow has taken 
the stance that technology, including Al, is intended to improve the lives of humans (i.e. “make the world of work, work better for people”) 
rather than disolace humans. Alin the workplace has caught significant attention and is at times perceived as negative rather than positive. 
While CIO's in organizations are promoting Al to intelligently automate work processes, many workers feel threatened that Al will eventually 
take over their jobs. 


Ethical considerations taken by ServiceNow include: 


Al Control: Al systems should perform the task intended by the developer. There should also be a mechanism for correcting unintended 
behavior. 


Al Safety: Al system should have built in safety mechanisms that reduce risks including privacy, abuse, security and unintended 
consequences. 


Value Alignment: Al systems should be designed so that their goals and behaviors can be assured to align with human values throughout 
their operation. 
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It will also allow people to learn new skills and focus on more efficient tasks. AI can help in better organization, reducing stressors and improve 
productivity. 
ace-can-poost-productivity 


avoue 


nito //sauciaazvetie.com sa/aricie/542 1 /S/TECHNOLOGY/A-in-the-Workp 


To demonstrate ServiceNow's thought leadership and ethical and thoughtful approach to this space, ServiceNow is designing the intelligent 
automation fo free up skilled employees for high-value work — increasing productivity, innovation and economic growth. This is described in 


the attached “Today's State of Work — At the Breaking Point” whitepaper { 


nitps://pwww.servicenow.com/content/cam/servicenow-assets/public/en-us/cdoc-ivyoe/resource-cenier/whnite-paper/sn-state-of-work-repori- 


The best practices and recommendations made within the document above relate to several products delivered with the ServiceNow 
platform. Each of the products include little to no opportunities for bias. Datasets are presented and predictions may be made, but 
ultimately humans make the decisions. 
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Selecting data records for training your solui 
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Exporting your solution for training 


Training Flow 
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